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Abstract 


This study developed a framework that can be used to build a forum environment, 
referred to as a Collaborative Interaction Management System (CIMS) that incorporates an 
automated method of assessing student’s individual contributions and group collaborative 
learning. The framework was fine-tuned by conducting a survey that identified 20 major 
features that faculty and students from various universities in the National Capital Region of 
the Philippines mostly prefer to be included in a forum environment. A web-based prototype 
software was then developed that implements the framework along with the assessment 
method and the 20 top features. To determine the validity of the main feature of CIMS (..e., 
the assessment method), sample forum data consisting of 28 threads with a total of more than 
1000 messages spanning various topic domains was collected from the Slashdot forum and 
the numeric ratings assigned to messages in this data was used to measure accuracy and 
reliability of the numeric ratings assigned by CIMS by analyzing it using Cohen’s Kappa. 
Based on the results of the survey, the overall means have identified that both faculty and 


students feel that the prototype software are functional, reliable, and usable enough. 
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1. Introduction 

In the traditional lecture-based method of teaching and direct instruction, learning is 
primarily perceived as involving the simple transfer of fixed knowledge from the teacher to 
the students. In this paradigm, the teacher's main responsibility is to cover as much course 
material as possible and deliver knowledge to their students. In such a setup, learning is 
conceptualized as a mere additive process where the minds of students are viewed as empty 
vessels in which knowledge can easily be poured into. As such, learning is simplistically 
presumed to have occurred when students have individually acquired the knowledge they 
received in their heads (Bloom, 1956). While such methodology and perspective has been 
around for a long time, and remains the most common form of instruction, it nevertheless has 
numerous disadvantages (Terenzini et. al., 2001; Baghcheghi et. al., 2011). 

Today, spurred by the development of several learning principles, approaches on 
classroom instruction have evolved considerably. Educators have increasingly realized the 
inadequacy of the lecture-based approach and have started to view learning as a process that 
is based more on student’s personal experiences. This realization has spurred the re- 
conceptualization of learning from an additive process to a developmental one wherein 
students are viewed to independently and continually re-organize knowledge and had created 
new representations for them. In this new paradigm, learning is defined as being context- 
sensitive and focused on self-discovery and exploration. Knowledge therefore is viewed as 
being created more within a group process where interaction between individuals and the 
sharing of experiences in the learning activities are the primary motivators. The teacher’s 
responsibility in this scenario is also shifted towards getting students to interact with each 
other in a productive and supportive manner rather than simply delivering course materials. 

In light of the need to develop discussion support tools that can help improve the 
utility of educational forum discussions, the proponent sustains the idea that an environment 
(similar to LMS and CMS) specifically suited for implementing online forums and managing 
student’s collaborative interaction are necessary precursors to promoting wide-spread 
utilization of this communication technology among higher educational institutions in the 
Philippines. In this research, such technology is referred to as Collaborative Interaction 


Management System (CIMS). The development of CIMS is the focal interest of this study. 
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The general objective of this study is to explore the feasibility of developing a CIMS 
environment suitable for use by teachers and students in the Philippine settings. Specifically, 
it aims to: 

1. Design a framework that can be used to develop the CIMS environment and 

implement it in a prototype web-based software artifact. 

2. Evaluate the accuracy and reliability of the performance of the system in 
providing assistance to teachers in managing the student’s collaborative 
interaction; and 

3. Evaluate the software quality of the prototype system in terms of the ISO 9126 


software quality characteristics. 


2. Theoretical Framework 


Jahng, N.; Nielsen, W. & Chan, E. (2010) provided a sample conceptual framework 


describing the new teaching paradigm as shown in Figure 1. 


Figure 1 
Conceptual Framework Describing a Collaborative Learning Paradigm 
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Source: Jahng, N.; Nielsen, W. & Chan, E. (2010) 
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Learning environments that emphasize the social nature of learning are considered as 
valuable tools in this new paradigm. Tools like forum discussions that pay particular 
attention to promoting and encouraging communicative cooperation and collaboration are 
good examples of this kind of environment. In fact, in the e-learning panacea, both in the 
academia and the industry, discussion forums are gaining tremendous popularity as a 
mechanism for increasing learner interaction (Rosenfeld and Gregory, 2012). 

Unfortunately, utilization of discussion forum in academic setting has been often 
problematic. The main culprit is the enormous amount of time and effort that it levies to 
implementing teacher-mediators in monitoring and guiding student interaction. Another 
factor is the high level of difficulty faced by the implementing teachers in evaluating the 
contributions of students in such online interactions (Roig and Rosales, 2012). In the field of 
eLearning, similar problems are being addressed through the development of specialized 
environments such as the Learning Management System (LMS) and Content Management 
System (CMS). 

The work described in this study aims to adopt the same strategy by designing a 
framework that could be used to develop a management system that could potentially provide 
teachers some assistance in managing the student’s collaborative interaction in forum 
discussions. Such environment can be referred to as a Collaborative Interaction Management 
System (CIMS). It also aims to propose a strategy for constructing and developing an 
application that implements and demonstrates the capabilities of the proposed framework. 

The conceptual model, shown in Figure 2, of the three important components of an 
Educational Asynchronous Online Discussion environment proposed by Raga (2013) 
provides the paradigm guiding this study. 

The model highlights the three components of an educational forum environment: (1) 
The Discussion Forum software, (2) The Interacting Students, and (3) The Teacher 
Moderator. It also shows that there is a problematic relationship that exists between each of 
these components. The model also highlights the importance of discussion support tools. 
These tools can provide functions that can act on the problems specified in order to neutralize 
or at least minimize its occurrence. These functions are represented as a, y, and B in the 


above model. 
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Figure 2 
The components of an educational asynchronous online discussion and the problematic 
relationship between them 
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Source: Raga (2013) 


For example, discussion support tools can provide information that can minimize the 
number of decisions that teachers have to make with regards the utilization of their time and 
attention and can also provide assistance with the assessment of individual student’s 
performance (y). Such functions can give teachers more flexibility in fulfilling their tasks as 
discussion moderators (a). At the same time, issues related to topic drifting and incoherent 
discussion can also addressed by allowing students some access to the interaction 
visualizations that automated support tools can generate (f). 

Based on the studies and insights taken from the foregoing concepts and theories, a 
conceptual model was developed to represent the elements of this study, their 
interrelationships, and boundaries. Figure 3 presents this conceptual model from a Systems 
(input, process, output) perspective. 

The conceptual model shows the different elements that define the overall scope of 
the objectives of the study. Each component corresponds to a distinct stage, which in turn, 


defines a set of requirements that the proponent needs to accomplish/develop. 
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Figure 3 
Conceptual Framework 
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Inputs to the study involved defining the various knowledge requirements of the 
problem that the proponent need to learn and understand thoroughly, including technologies 
that need to be adopted as well as the characteristics of the would be stakeholders of the 


study. 
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Processes in the conceptual model pertain to the research activities that need to be 
conducted by the proponent in order to address the research questions. This covers the 
conceptualization of the system design, methods for system development, strategies for 
assessing the performance of the system, and tools for evaluating the quality of its 
characteristics. 

The outputs of the study were considered from a short-term, long-term, and impact 
point of view. The short-term output was the development of a fully functional CIMS 
prototype that demonstrates the feasibility of the idea proposed in this study. Aspects of the 
long-term solution, in turn, would involve the promotion of a monitoring/mediating model 
specifically suited for the online forum learning environment. Finally, the impact of the study 
on society as a whole is its projected effect on the improvement of quality education through 


consistent utilization of Collaborative Learning and Outcomes-Based Education. 


3. Methodology 
3.1. Design Issues 


The first objective was addressed by means of a literature review focusing on the 
characteristics of different designs of online forum discussion applications. By drawing on 
the standard characteristics of these applications, a questionnaire was developed that was 
used to extract opinions from faculty and students from different universities to determine the 
best features that can be incorporated into the design of CIMS. Furthermore, an open 
question was posed for the respondents to collect suggestions on other non-standard features 
that may improve the design configuration of the system. The survey data were processed in 
terms of descriptive statistics while content analysis were used to summarize responses to 
open questions. 

Due to financial constraints, however, only a limited number of universities were 
accommodated in this study. These consisted of several universities from the National 
Capital Region area, including the university where the respondent teaches. The other 


universities were selected based on their proximity to the respondent’s university. 
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3.2. System Prototype Development 
In relation also to the first question, figure 4 depicts the outline that was followed in 


the development process of the system. 


Figure 4 
Outline of prototype system development process used in this study 
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In general, as the review of literature revealed, problems associated with 
implementing educational forums can be summarized into two key issues: (1) Discussion 
moderation, and (2) Assessment and Feedback. As such, at the most fundamental level, the 
purpose of the CIMS environment is to monitor incoming message contributions of students 
in order to track the collaborative activity of student groups and automatically provide 
assessment reports on learning progress to moderating instructors. The context diagram in 
Figure 6 outlines some of the salient details illustrating the general applicability of the CIMS 
environment for this purpose. 

For the activity monitoring to commence, the mediating instructor must first provide 
the system with reference texts/learning materials for the topic to be discussed along with 
other discourse parameters. The reference texts will be used as background domain 
knowledge by the system in generating numeric ratings of the topical relevance of each 
message contributions. 

The numeric ratings of the message contributions, in turn, will be used to estimate the 
domain knowledge state of individual students, which will also form part of his/her 
participation profile. The participation profile will be made accessible to all students in order 
to ensure immediate participation feedback, allowing students to independently adjust their 


participation level/pattern. 
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Figure 5 
Proposed architecture of the CIMS environment 


Reference and 
discourse 
parameter 


databace 


Domain Knowledge 


Estimated Topic Hierarchy 


student 
knowledge 
state 


comeas 


Collaborative 
{___Leorning __ 
Assessment 
Renortc 


Message Content and 
Hierarchy 


Reference Text (s) and 
other discourse 
parameters 


Student 
participation 


profile 
= 


immediate 
Feedback 


Moderating 
Instructor 


Mediation 


Discussion 


transcript 
Message 
—_—_—_—— 
Contribution 
Student Post (s) © 


Source 


Recipients 
(Group 
Members) 


The technical skills utilized by the proponent that has an impact on the development 
of the CIMS environment included the following: 
e programming skills for Java Server Pages (JSP), Javascript, and HTML; 
e administrative knowledge of webserver, more particularly, resin-server 
e database design and implementation using Microsoft Access 
Utilization of these tools is described as follows: 
e HTML (Hypertext Markup Language) was used to develop the front-end Web pages 
of the system 
e JSP (Java Server Pages) was used to create a dynamic Web page interfaces that 
interact with the database; 
e Javascript was used to write client-side validation scripts 
e Resin Server was used as a proxy web server. 
e The appropriate database structure was created to hold the system data and 
implemented in .mdb format 


e Support data was stored either in XML format. 
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Figure 6 
High level system overview of the system design 
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Due to the short span of development time allotted to this study, the design and 


behavior of the system was tested using only alpha testing and limited beta testing. To 
evaluate the system’s compliance with the specified requirements and functionalities, the 
alpha testing was done as follows: 

e Testing of all hyperlinks in the system Web-based interface; 

e Entering of sample data via the Web-based interface to test database functionality and 
system displays; 

e Testing of all input fields for JavaScript validation where relevant. 

Two sets of forum transcript data will be used as dummy inputs follows: 

1. Malfunction dataset. This dataset will be intentionally designed so that it may cause 
the system to malfunction or return incorrect information. The purpose of this is to 
cull out any unseen bugs in the system in order to fix it and improve the ability of the 
system to adapt and respond to incorrect input. 

2. Performance dataset. This dataset will be designed to measure the system’s 
implementation of the assessment methods along with the individual processes and 
data flows. 

3.3. Assessment Issues 
In order to address the second objective, the approach developed by Raga (2013), a 


word space-based approach for measuring the topic relevance of message contributions in 
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asynchronous discussions will be adopted in this study. The approach implements an 
algorithm that combines two of the most important text signatures used in the representation 
of word semantics, (1) the lexical distribution similarity and (2) the statistical word 
regularities, in order to generate the topical relevance ratings of individual messages. Figure 
7 specifies the basic schematic of this approach. 

The approach consists of a two-stage process: (1) the training phase, and (2) the 
testing phase. The training phase requires a reference document input to be used for 
generating background domain knowledge of the system while the training phase processes 
incoming forum message contributions and measures the semantic proximity of its contents 
of the current domain knowledge of the system. Both phases heavily rely on Natural 
Language Processing Technologies (NLP). 

The message ratings are addressed to the mediating instructor, wishing to distinguish 
the important contributions of a forum, for further examination, thus helping him/her to 
diminish the workload required with traditional methods (e.g. reading all the messages 
sequentially). Thus this approach is suitable for use in large forae, containing many threads, 
produced by many users, similar to the environment which is the context of this study. 


Figure 7 
The word-space based approach proposed by Raga (2013) for measuring the topic relevance 
of message contributions 
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The word-space-based approach enables any system to visualize the topic structure of 
any forum discussion. However, it only generates numeric ratings for individual messages. 
For this reason, to model the collaborative learning interaction of students and generate 
numeric group ratings, the proponent suggests the following formula which simply averages 
the numeric ratings of all the messages over the total number of messages generated in a 
group. 

A word-space model is a spatial representation that derives the meaning of words by 
plotting these words in an n-dimensional geometric space (Sahlgren, 2005). The process of 
plotting words in a word-space is similar to the way points are plotted in a two dimensional 
graphing paper. The main difference is that, in the case of a word-space, the dimension n can 
be arbitrarily large; the size of which is determined by the number of unique word type in the 
set of words to be plotted. 

Usually, the coordinates used to plot each word depends upon the frequency of the 
contextual feature that each word co-occur with within a text. For example, words that do not 
co-occur with the word to be plotted within a given context are assigned a coordinate value 
of zero. The set of zero and non-zero values corresponding to the coordinates of a word in a 
word-space are recorded in a so-called context vector. Because most of the words in any text 
dataset will never co-occur with a particular word, the coordinates recorded in a context 
vector will often be sparse or full of zero values. 

By itself, the position of a word in a word space does not indicate anything about its 
meaning. To deduce a certain level of meaning, this position needs to be measured relative to 
the position of other words. In this sense, a linguistic concept known as the “Distributional 
Hypothesis” is applied. This concept states that: “words that occur in the same contexts tend 
to have similar meanings”. Having similar contexts means that words are surrounded or that 
they co-occur with same set of words, thus, if we plot these words in a word-space they 
would be positioned close with each other. The level of closeness of words in the word-space 
is often referred to as the spatial proximity of words. This spatial proximity is what is used to 
represent the semantic similarity of words. A common approach used to determine spatial 
proximity is to measure the cosine of the angle between the context vectors; this approach is 


known as the cosine similarity measure. The formula for computing cosine is as follows: 
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LW jWij 
j= W290; j= W? ij 


Where: Q is a vector representing one term or document, D is a vector representing 


Equation 1 CoSim(Q, D;) = 


another term or document related to Q, and w are term weights 

Currently, there are three major approaches to implementing a word space. These 
include Latent Semantic Analysis (LSA), Hyperspace Analogue to Language (HAL), and 
Random Indexing (Sahlgren, 2005). For our purpose, we opted to use the Random Indexing 
approach. 


n 
Equation 2 Umnsgaa titimsg 


n 


Where: 
inr is the topical numeric rating of each message 
n is the total number of messages in the discourse transcript 

Since this formula hinge on the numeric ratings generated by the word-space-based 
approach, validating its efficiency also hinges on the efficiency of the measures generated by 
the word-space-based approach. However, the only means of measuring the efficiency of any 
automated system is to compare it with the decisions generated by human analysis. Such 
inquiry will no doubt require a huge amount of data transcript, not to mention, manpower. 

To go around this issue, this study resorts to experiments that compare the 
performance of the system on educational discourse transcripts downloaded from the 
Slashdot forum. Slashdot is a popular online technology-news and discussion site with a 
large membership. Its main feature is its community based moderation system that awards an 
explicit rating of between -1 and +5 for every message posted in its threaded discussion, 
where posts considered as less interesting are rated -1 and highly interesting and quality posts 
are rated 5. Studies show that this rating system can be used to separate particularly good and 


bad content from the average ones. 
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Figure 8 
The SlashDot Discussion Board 
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The message data from SlashDot was considered as providing a suitable proxy data 
for the assessment of CMIS’s performance for two reasons: 

1. The format of the Slashdot forum is fairly similar to an educational discussion type of 
interaction where a reference document is provided and the participants are asked to 
align the topic of their contributions to the topic represented by the reference 
document. 

2. The ratings that Slashdot assigns to each message is fairly synonymous to a relevance 
rating since it (Slashdot) also uses these ratings to implement a recommender and 
reputation feature for participants that only want to read selected messages. This 
means that the message ratings can be interpreted and used as code tags (e.g., 
Relevant or Not Relevant) for the messages and this code tags can be compared to the 
ratings generated by CMIS. 

A total of one thousand messages from SlashDot will be downloaded for this purpose 


and Cohen's kappa (Cohen, 1960) will be used to compare the decision of the system and the 
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ratings of assigned by SlashDot. Cohen's kappa (K) measures the agreement between two 


raters who each classify N items into C mutually exclusive categories. The equation for K is: 


: _ P(A)-P(E) 
Equation 3. K = Pe) 


Where: P(A) = the number of judgments on which the coders agree 
P(E) = the number of judgments for which agreement is expected by chance. 
After computing for the value of K, the magnitude guidelines shown in Table 1, as suggested 


by Landis and Koch (1977) can then be used to interpret this kappa value. 


Table 1 
Interpretation of the magnitude of Kappa values as suggested Landis and Koch 


Kappa (K) Strength of Agreement 
< 0.00 Poor 

0.00 — 0.20 Slight 

0.21 — 0.40 Fair 

0.41 — 0.60 Moderate 

0.61 — 0.80 Substantial 

0.81 — 1.00 Almost Perfect 


To give an example of how Cohens Kappa is used. Consider a scenario where two 
coders are tasked to classify N items into one of two possible categories (e.g., Relevant (R) 
or not relevant (NR)), the format of the contingency table that will be generated will be as 


shown in Table 2. 


Table 2 
Format of the contingency table used in computing Cohen’s Kappa 
Coder 2 
NR R Total 
NR A B G=A+B 
1 
Coder! amc D H=C+D 
Total |; E=A+C |F=B+D_ |N=A+B+C+D 
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In this table, A is the number of items that both coders classified into the NR 
category, while B is the number of items that Coder 1 classified into the NR category but 
Coder 2 classified into the R category, etc. With this contingency table, computing for P(A) 


and P(E) are as follows: 


(A+D) 


Equation 4. P(A) = = 


and 


Equation5. P(E) = (=) (=) + (=) (7) 


To get an idea of how the kappa statistics work, consider the contingency tables 


provided in Table 3. Both tables describe the classification decisions of two coders on a 50 


items set. 
Table 3 
Sample contingency tables for computing Cohen's Kappa 
Coder2 
NR R 
NR | 2 
Coder] R a 37 
Contingency table #1 
Coder2 
NR R 
NR 16 6 
Coder] R 8 0 
Contingency table #2 
Computing for the kappa of the two coders in the first contingency table, we get: 
(2 + 37) 
P(A) = ——— = 0.78 
(A) =O 
6\(9 44\ (41 
PCE) (=) (Gs) * Ga) Ge) 
P(E) = (0.12)(0.18) + (0.88) (0.82) 
P(E) = 0.022 + 0.722 
P(E) = 0.744 
_ (0.78 - 0.744) 
1 0.744 
K = 0.14 
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On the other hand, if we compute for the kappa of the two coders in the second 


contingency table, we get: 


P(A) = Ct) 269 


50 
BY) Gs) Ga) ea) 
P(E) = (0.12)(0.18) + (0.88)(0.82) = 0.5 


_ (0.72- 0.5) _ sae 
~ 4 — 05° ~ 


4. Findings and Discussion 

4.1. Survey Results — Selecting Features for the CIMS Environment 

In order to select features to be incorporated into the design of SIMS, surveys were 
sent out to faculty and students in four universities, namely FEU-East Asia College, San 
Sebastian College, Far Eastern University, and University of Sto. Tomas. A total of 40 
faculty members and 80 students responded to the survey. 

Table 4 shows the results of the survey indicating the preferences of the faculty and 
student respondents. As anticipated, a wide variety of features were highly selected by the 
participants. Features which garnered the highest mean scores are shown in Table 4. 

While the respondent’s initial preferences was captured by the feature categories 
included in the questionnaire, there are possibly other specific features that the respondent’s 
may have in mind. By posing an open question at the end of the questionnaire, the 
respondents were given the freedom to list their preferences towards non-standard features 
that may also improve the design of the CIMS configuration. Unfortunately, no responses 
were collected from the participants in this category. 

Obviously, due to limited resources and development time, not all the features found 
in the literature can be programmed into the design of the CIMS environment. However, 
these 20 features will be incorporated into the CIMS prototype to demonstrate the feasibility 
of developing a low-cost, extendable, customized forum environment. In addition, to address 
the limitations of conventional forum systems found in the literature, the majority of the 


respondent’s development effort was focused on building the main feature of CIMS, that is, 
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the feature that focuses on assisting teachers in moderating and assessing student’s 


interaction. 
Table 4 
Forum features which garnered the highest mean rating in the survey 
Feature No. / Feature Description N Mean 
1 56. Log new registrations including date/time 120 4.27 
2 15. Report offensive or unwelcome post 120 4.26 


49. Moderate member’s uploads: view, download and delete 
using the built-in File Manager 


4 10. Topic Rating System 120 4.22 
4. Topic display options - Sort by Topics, Topic Starter, 


120 94.23 


: Replies, Views, Post Date, etc. pe —— 

6 3. Message Searching Features using Title, Description, and 1200 4.19 
Keywords 

7 55. Log Topic/Post Deletion, Hiding, Moving, Editing, etc. 120 4.19 

8 17. Admin tools for Posts - Edit, Delete, Move, etc. 120 4.18 

9 36. Forum Statistics 120 4.17 

10 27. Paste From Word Tool 118 4.16 

11 42. Member FAQ/help files on how to use the forum 118 8 4.15 

12 47. Search for member's posts 119 4.15 
48. Moderator permissions - Moderators can Edit and Delete 

13 ; 120 3 =4.15 
Topics and Posts, and can also suspend members 

14 18. Delete Posts - If a member has permission they can only 119 4.13 
delete their post if a reply has not been posted ; 
44. Member Information - DOB, Age, Location, Homepage, 

15 : 120 =4.13 
Occupation, Interests, etc. 

16 46. Active Statistics - Online Status, Join Date, Last Visit, 120 413 
Number of Posts, etc. 

17. 54. Select what actions you want recorded in log files 120) =4.13 

18 26. Spell Checking Support 118 4.11 

19 29. Member List with Search Tool 120 34.10 

20 37. Edit member profile 120 34.10 
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4.1.1. The CIMS Software Prototype 

Viewed from the perspective of its main feature, the system is composed of five 
blocks: the Text Processing, the Rating Generator, the Profile Generator, the Group 
Assessment Generator, and the main Coordinator module. Figure 9 shows the block diagram 


of the system and the items below describe the corresponding modules and interfaces. 


Figure 9 
Low Level diagram of the CIMS main function 


Student 
Source 


; Text 
Immediate 


Feedbackto | Student Profile 


Processing 
students Generator 


Ratings Message content 
Moderating and Hierarchy 
Instructor 


Collaborative 
Learning 
Assessment 


Coordinator 


Group’s Learning 
Assessor 


Domain Knowledge /Topic Hierarchy 


Message Contribution 


Almost the entire system (approximately 98% of all functions) was developed in JSP. 
The data manipulated by the system, i.e. Discussion text, participant’s data, reference texts 
and message representations, are stored in a relational database using the .mdb format. 
Supporting data used by the word-space based algorithm, in turn, is stored in XML files. As 
the files are read, corresponding data structures and representations are created and loaded 
into memory by JSP functions. The web-based interface, which is responsible for the 
system’s dynamic interaction with the users in the discussion forum environment, was 
developed using combinations of JSP, HTML, and Javascript. Figure 10 shows the homepage 


interface of the CIMS prototype environment 
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Figure 10 
The CIMS Homepage 
Collaborative Interaction Management System (CIMS) - Beta 
oe o 
e CIMS-Beta is a prototype discussion board platform. It is designed to test an akternative discussion environment where every contributions 
posted in the discussion are rated and tracked by an automated assessment method. The analysis engine of the assistive technology of this 
hi beta is powered by a Word-Space based algorithm. CIMS-Beta is a product of the research work conducted by Salvacion Mata-Domingo 
Copyright @ SM-D, LLC 2014. All Rights Reserved 
C.1.M.S: pee 
2) Home '@ Login !¢! Register '=) Enrol 


Today is Wednesday, May 21, 2014 (1:32 PM) 
Your currently logged-in as a guest 


To participate in the discussion forum you need to 


Forum Details 
Registered users have posted a total of 150 messages 


e There are 10 topic of discussions 
11 users are currently registered 


The newest registered user is Menchie 


C.].M.S. 3 user(s) are currently enine 
Registered users: 


LFeatures | Rules | Credits | Help | Admin | 


The Coordinator Module 

The coordinator module acts as the glue of the system. It’s the source of data for all 
the other modules because it manages the interface that implements the discussion 
environment enabling CIMS to collate message contents and its hierarchy. 

In addition, the coordinator module also handles and activates a large number of 
lateral functions, which represents the various features incorporated into CIMS, as illustrated 


in figure 11. 


Figure 11 
Varied Functions of the Coordinator Module 
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Many of these functions were designed to be conveniently accessed by the teachers 


through the CIMS Admin Dashboard shown in figure 12. 


Figure 12 
CIMS Admin Dashboard Interface 


Collaborative Interaction Management System (CIMS) - Beta 
e CIMS-Beta is a prototype discussion board platform. It is designed to test an altemative discussion environment where every contributions 
vA posted m the are rated and tracked by an automated assessment method. The analysis engine of the assistive technology of this 
beta is powered by a Word-Space based algorithm. CIMS-Beta is a product of the research work conducted by Salvacion Mata-Domingo 
Se I M Ss Copyright @ SM-D, LLC 2014. All Rights Reserved 
2) Home (©) Profile || Admin Sl Search [=| Membertst ') Logout < Sally > 


CIMS Admin Dashboard 


ie 


: 
ext Tope References 9 view Foun Ratings 


Features | Rules | Credits | Heip | Admin | 


The GUI functions. These are the functions responsible for the low-level processing 
of text contributions such as adding/deleting/editing messages and managing the list of 
participants. 

The Discussion functions. Functions that perform all actions required to launch a 
discussion, i.e. To distribute the issues among the participants and to build the initial 
configuration of the discussion environment. 

The Lateral functions. These are functions that are not necessarily essential to the 
conduct of the forum discussion, but provide additional convenience to the users such as 
Spell Checking, Topic Display Options, Forum Statistics, Searching, etc. 

The Membership functions. These are functions responsible for collecting and 


maintaining profiles of participants. 
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The Logging functions. These are functions that implement security in the system and 
the update of maintained log files. 

The Low-level functions. These are the functions responsible for the low-level 
processing of text contributions such as adding/deleting/editing messages and managing the 
list of participants. 

The Text Processing Module 

In order for CIMS to produce automated ratings of forum messages it needs a 
diagnostic engine that can analyze the text contents of each message. The analysis process, 
among other things, is responsible for manipulating the text component of messages and 
makes use of NLP technology to extract descriptive statistics from these texts and passes 
these to the rating generator module. 

The Rating Generator Module 

In order to generate ratings for the messages, the Rating Generator module compares 
the text statistics of the contents of each message to the overall text statistics generated in the 
reference text provided by the instructor. The measured amount of similarity is treated as the 
pedagogical value of the message. It then uses the formula provided in (2) to generate 
corresponding learning assessment for the group. Figure 16 displays the interface for 
converting text data into word space values which is used to compute for the ratings of 
the ,messages. This interface is available to the teachers through the admin interface of 


CIMS. 


Figure 13 
CIMS Interface for transforming text data into wordspace values, the first step for the rating 
generator process 


‘Collaborative Interaction Management System (CEMS) - Beta 
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After the appropriate word-space have been generated, the second step of the rating 
process is to convert each message into its equivalent word space form and compare it 
numerically to the word-space of the reference text. The comparison is run through using the 
cosine-similarity measure which generates a value between -1 and |. The values generated 
represent the degree of similarity between the reference text and the message contents. Figure 
14 shows the interface for this process and it represents the second step of the rating 


generation process. 


Figure 14 
The second step of the rating generation process. Comping the cosine similarity between the 
word space equivalent of the reference text and each message. 

Collaborative Interaction Management System (CIMS) - Beta 


posted in the discussion are rated and tracked by an automated assessment method. The analysis engine of the assistive technology of this 


eS CIMS-Beta ts 2 prototype descussion board platform It ts designed to test an alternative discussion envronment where every contributions 
| beta is powered by a Word-Space based algorithm CIMS-Beta is a product of the research work conducted by Salvacion Mata-Domingo 
C.1.M.S. 


Copyright @ SM-D, LLC 2014. All Rights Reserved 
iHome #) Profite (2| Admin © Search (2| Membertist Logout < Salty > 


Total of 14 messages extracted 


Details of Paradigmatic Analvsis Generated 
Forum to be analyzed : PowerVR Discussion Group 


The Student Profile Generator Module 

The student profile generator is responsible for the low-level manipulation of the 
message ratings to generate a representation of the student’s comprehension of the topic of 
discussion; this representation is referred to as the student profile. Figure 15 shows the main 


interface for generating a student profile. 


A Publication of the Institute of Industry and Academic Research Incorporated 
www.iiari.org 


INTERNATIONAL JOURNAL OF INFORMATION AND COMMUNICATION 
TECHNOLOGY AND EDUCATION 


Volume 1, Issue 1 - September 2020 - ISSN 2719-065X (PRINT) 2719-0668 (ONLINE) 


Figure 15 


rete for scene student profile 
Collaborative Interaction Management System (CIMS) - Beta 


CIMS-Bets is a prototype discussion board platform. It 1s designed to test an alternative discussion environment where every contributions 

posted m the descussion are rated and tracked by an automated assessment method. The analysis engine of the assstive technology of this 

beta is powered by a Word-Space based algorithm CIMS.Beta is 2 product of the research work conducted by Salvacion Mata Domingo 
CLI MLS. Copyrtt  nt0, 110 2014 An Rights Reserved 

2 |Home (©) Profite |2|Admin (Search |2| Memberlist '>\Logout < Raven > 


Select Forum to View: Select Forum Here 
Recah cians Cece Fs Rea a Rated ONE DES Coa) 


CHART OF OVERALL ANALYSIS OF POSTED MESSAGES 
Forum: PowerVR Discussion Group Student Raven Total # of Message Contributions: 6 


Profile of Student Raven | Total = of Messages Posted: 6 | Forum: PowerVR Discussion Group 


he mobile graphics space has been dommated by one player: PowerVR. ARM and nVidia are more 


cent entrants. AMD doesnt yet have anything in this space, although that will change very soon. 0.33759 


In a nutshell, the process of generating the student profile involves collecting all the 
highly rated content-bearing keywords in the reference text and assigning a numeric value to 
each by comparing the context of how it is used in the message as oppose to the context of 
how it was used in the reference text. The teacher is responsible_for_configuring/selecting 
the final set of keywords to be used as profile templates of the students. Figure 15 shows a 
sample student profile generated by CIMS including the graphic representation of this profile 
automatically generated by CIMS. 

The representation serves the purpose of providing immediate feedback to the 
students when interacting with each other, allowing them to self-assess and dynamically 
adjust their level of participation independent of their instructor. 

The Group Learning Assessor Module 

Along with the individual message ratings, CIMS was also designed to generate 

group ratings representing the perceived collaborative learning accomplishment of the group. 


The formula described in (1) was used to compute these ratings. 
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Figure 16 
Summary of Group Appropriacy 
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Figure 16 shows the CIMS interface for activating the process for group assessment 


while figure 17 shows a sample output of this process. 


Figure 17 
CIMS Interface for activating the group assessment process 


Collaborative Interaction Management System (CIMS) - Beta 
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Figure 18 


Sample output of the group assessment process 
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4.2. The accuracy and reliability of the performance of the system 

The validity of the assessment analysis generated by CIMS depends on the quality of 
the numeric ratings assigned to each message by the rating generator module. As already 
pointed out in previous discussions, the task of measuring the quality of numeric ratings 
automatically generated by a system can only be accomplished by comparing it to the 
decisions rendered manually by human judges. This, however, is not an easy task for several 
reasons. Firstly, a huge bulk of data must be used to ensure sufficient amount of examples. 
Secondly, there is the task of asking the human judges to read each message manually in 
order to render their judgment. 

As such, in order to go around these problems, the assessment strategy adopted is to 
simply compare the ratings generated by the system to the ratings assigned to messages in the 
Slashdot forum. The objective of this strategy is to determine whether the numeric ratings 
assigned by CIMS will match the actual ratings assigned by human moderators of the 


SlashDot forum. 
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Table 5 summarizes the performance dataset used for this purpose. The number of 


discussion threads extracted from the SlashDot Forum along with the Thread topic and the 


number of messages extracted for each thread are also provided. 


Table 5 
The performance dataset for the SlashDot experiment 


Thread 
Topic 


PowerVR 
Google 
WinPhone 
Android 
SmartPhone 
Crawler 
MSSneakAttack 


Streaming 
SETI 
ObamaRomney 
Cyborg 

Avatar 
Mandriva 


Fuelbank 
XBoxAd 
Ubuntu 

Robots 
SpamBotNet 
Microsoft 
ClimateScience 
WifiSniffing 
Cloud 


Patent 


ArchLinux 
PhonyLaser 


Estonia 
StemCells 
Amazon 


# of 
msgs 


19 


25 


42 


39 


25 


23 


32 


66 
78 
57 
31 
35 
31 
61 


46 


51 


41 


53 


48 


17 


20 


70 
52 
31 


53 


16 
33 
55 


Forum URL 


http://hardware.slashdot.org/story/12/06/17/155 1212/powervr-to-make-mobile-graphics-gpu- 
compute-a-three-way-race-again 
http://yro.slashdot.org/story/12/06/18/0438207/us-govt-demands-for-google-data-up-37-over- 
the-last-year 
http://tech.slashdot.org/story/12/06/21/2015209/will-microsoft-extend-surface-model-and- 
manufacture-windows-phones 
http://hardware.slashdot.org/story/12/08/25/2027233/can-android-revolutionize-spacecraft- 
design 
http://hardware.slashdot.org/story/12/07/15/2139248/georgie-smartphone-for-the-blind-and- 
visually-impaired 
http://science.slashdot.org/story/12/09/07/1457227/nasas-giant-crawler-transporter-is-getting- 
an-upgrade 

http://apple.slashdot.org/story/12/08/23/17522 16/os-x-108-vs-ubuntu-on-apple-hardware- 
benchmarked#comments 

http://entertainment.slashdot.org/story/12/06/19/153424 1/young-listeners-opt-for-streaming- 
over-owning 

http://science.slashdot.org/story/12/07/03/220220/seti-running-out-of-money 
http://politics.slashdot.org/story/12/09/04/1943209/obama-and-romney-respond-to- 
sciencedebateorg-questionnaire 
http://hardware.slashdot.org/story/12/08/29/2232253/harvard-creates-cyborg-tissues 
http://hardware.slashdot.org/story/12/07/07/2249235/fmri-lets-israeli-student-control-robot- 
in-france-with-his-mind 
http://linux.slashdot.org/story/12/07/09/1648212/mandriva-juggles-multiple-codebases 
http://hardware.slashdot.org/story/12/08/29/2223207/is-an-international-nuclear-fuelbank-a- 
good-idea 
http://games.slashdot.org/story/12/07/13/1924238/the-ugly-profitable-details-about-xbox- 
live-advertising 
http://apple.slashdot.org/story/12/08/23/1752216/os-x-108-vs-ubuntu-on-apple-hardware- 
benchmarked#comments 
http://hardware.slashdot.org/story/12/09/04/2034225/social-robots-may-gain-legal-rights- 
says-mit-researcher 
http://news.slashdot.org/story/12/07/05/2232209/microsoft-engineer-discovers-android-spam- 
botnet-google-denies-claim 
http://it.slashdot.org/story/12/07/12/166200/microsoft-kills-windows-gadgets-via-security- 
update 

http://news.slashdot.org/story/12/07/15/16452 19/entangled-histories-climate-science-and- 
nuclear-weapons-research 

http://tech.slashdot.org/story/11/07/01/163721 1/judge-oks-wiretap-lawsuit-over-google-wi-fi- 
sniffing 

http://idle.slashdot.org/story/12/08/29/22 15244/survey-reveals-a-majority-believe-the-cloud- 
is-affected-by-weather 
http://mobile.slashdot.org/story/12/09/05/1935208/google-awarded-face-to-unlock-patent 
http://apple.slashdot.org/story/12/08/23/1752216/os-x-108-vs-ubuntu-on-apple-hardware- 
benchmarked#comments 

http://hardware.slashdot.org/story/12/08/22/2 15222 1/phony-laser-security-system-proves- 
perception-is-reality 

http://developers.slashdot.org/story/12/09/04/2 136256/estonia-to-teach-programming-in- 
schools-from-age-6 
http://science.slashdot.org/story/12/09/12/2035226/stem-cells-turn-hearing-back-on 
http://hardware.slashdot.org/story/12/08/21/1419203/amazon-wants-to-replace-tape-with- 
slow-but-cheap-off-site-glacier-storage 
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Table 6 below provides a summary of the results generated for this test. Notice that in 
many of the datasets, the system was able to achieve acceptable Kappa performance using as 
few as two documents and with the size of these documents being well within the capacity of 


human teachers to collect. 


Table 6 
Summary of the results of the SlashDot Tests. 
# of Total size 

sa Accuracy Precision Recall Kappa _ ref. of ref. 

opic# 

docs docs used 

1 0.95 1.00 0.94 0.77 3 1,732 
2 0.92 0.90 1.00 0.71 2 1,176 
3 0.86 0.88 0.88 0.70 13 5,373 
4 0.85 0.93 0.72 0.69 ) 1,745 
5 0.84 1.00 0.76 0.68 7 2,941 
6 0.87 1.00 0.83 0.68 7 3,387 
7 0.84 0.90 0.86 0.66 6 3,170 
8 0.93 0.93 0.96 0.63 6 2,597 
9 0.82 0.87 0.83 0.63 6 3,942 
10 0.84 0.88 0.90 0.63 9 4,239 
11 0.81 0.69 0.92 0.62 5 7,735 
12 0.80 0.70 0.93 0.61 6 2,711 
13 0.81 0.75 0.94 0.61 6 2,342 
14 0.84 0.86 0.90 0.61 4 2,666 
15 0.89 0.97 0.90 0.60 8 4,224 
16 0.80 0.77 0.89 0.60 4 1,938 
17 0.78 0.78 0.88 0.58 3 4,558 
18 0.83 0.91 0.85 0.57 1 3,959 
19 0.79 0.58 0.85 0.54 3 836 
20 0.81 0.83 0.91 0.54 3 1,144 
21 0.81 0.87 0.87 0.53 2 856 
22 0.79 0.77 0.93 0.51 11 7,084 
23 0.73 0.61 0.90 0.48 6 5,054 
24 0.74 0.74 0.82 0.47 5 3,102 
25 0.72 0.61 0.91 0.46 2) 2,643 
26 0.69 1.00 0.58 0.41 1 336 
27 0.70 0.62 0.87 0.41 5 2,353 
28 0.71 0.74 0.83 0.34 4 6,022 


The results of this test, to some extent, provided evidence of the ability of the 


prototype software to generate acceptable results. In addition, the experiment also offered 
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some very interesting insights into the characteristics of the system’s performance. The 
results, for example, revealed some evidence that increasing the amount of training data also 
increased the reliability of the systems decisions if the additional training data also focused 


on topics covered by some of the messages. 


Novelty of the Study 

Currently, available discussion forum systems lend support to on-going discussions 
by merely storing messages and organizing the discussion transcript in some form of 
hierarchy (e.g., threaded or linear). Although these structures provide distinct advantages and 
have gained considerable success, they also tend to promote poor results from group 
interactions (Romiszowski, 1995) and levy considerable effort to teachers in identifying and 
mediating dispersed learners. 

The study aims to provide a framework that can be used to extend the capabilities of in- 
place structures of current forum systems for supporting online discussions. The framework 
will need to incorporate features that monitor the discussion and alleviate the task of 
facilitating and mediating it by providing both the teacher-mediator and the student- 
participants with an automated means of measuring the quality of posted messages. This 
quality measurement in turn will serve as a basis for generating continuous evaluation that 
will provide: 

(1) Hints to the student-participants, during message production, on the level of relevance 

of their contributions to the topical direction of the discussion 

(2) Feedback to teacher-mediators regarding the current status of each individual 

student’s exerted level of participation as well as the overall quality of the interaction. 

For the teacher-mediator, the feedback will provide a clear indication of problems that 
could develop within the discussion atmosphere ahead of time. For the student-participants, 


the hints during message production will provide self-assessment capabilities. 


5. Conclusion 
Upon detailed review, it became evident that the biggest weakness of conventional forum 


applications is their lack of support for instructors’ moderation and assessment of a student’s 


A Publication of the Institute of Industry and Academic Research Incorporated 
www.iiari.org 


29 


INTERNATIONAL JOURNAL OF INFORMATION AND COMMUNICATION 
TECHNOLOGY AND EDUCATION 


Volume 1, Issue 1 - September 2020 - ISSN 2719-065X (PRINT) 2719-0668 (ONLINE) 


interaction (Raga, 2013). Therefore, this study developed a framework that can be used to 
build a forum environment, referred to as a Collaborative Interaction Management System 
(CIMS) that incorporates an automated method of assessing student’s individual 
contributions and group collaborative learning. The framework was fine-tuned by conducting 
a survey that identified 20 major features that faculty and students from various universities 
mostly prefer to be included in a forum environment. A web-based prototype software was 
then developed that implements the framework along with the assessment method and the 20 
top features. Although only 20 features was incorporated, and not all the possible features of 
a forum system identified in the literature, in the design of CIMS due to lack of time, the 
overall architecture of the prototype was meant to be modular and expandable. This means 
any other feature that can easily be incorporated into it in the future. As such, the main 
difference between CIMS and conventional forum systems is its ability to provide assistance 
to teacher’s implementing online forum discussion as a pedagogical tool. 

To determine the validity of the main feature of CIMS (i.e., The assessment method), 
sample forum data consisting of 28 threads with a total of more than 1000 messages spanning 
various topic domains was collected from the Slashdot forum and the numeric ratings 
assigned to messages in this data was used to measure accuracy and reliability of the numeric 
ratings assigned by CIMS by analyzing it using Cohen’s Kappa. Using Grove et al’s (1981) 
acceptable criteria of K > 0.5, results show that CIMS was able to achieve acceptable 
consistency with a Kappa rating K>0.5 among 78.57% of the threads while it was able to 
achieve the standard the standard acceptable kappa K>0.6 among 57.14% of the threads. 
Although not perfect, these figures indicate that the numeric ratings generated by the system 
can be dependable to some degree. 

Finally, while establishing the consistency of the assessment methods of the system, there 
is also a need to determine the quality level of its software characteristics. While the ISO 
9126 software quality model defines several criteria to follow, the nature of the CIMS 
prototype dictates that Functionality, Reliability, and Usability were the primary criteria that 
is easily assessable by its user groups. In other words, these are the criteria that can be 


adopted to enable faculty and students to assess the prototype software. Based on the results 
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of the survey, the overall means have identified that both faculty and students feel that the 


prototype software are functional, reliable, and usable enough. 


The CIMS prototype software is by no means perfect. Nor can it be designed to be so 


because the deadline and limited resources of the study make it impossible to implement 


optimized applications: 


L, 


During testing, it was found that the procedure for processing text data inside the 
system proved to be a lengthy process and may not be feasible for implementation for 
medium-large sized classes. It is recommended that research can be done to try and 
improve the procedure by minimizing overheads processes. 

There is also plenty of space for optimization with regards to the time and space 
complexity requirements of the system. At the moment, based on experience, running 
the assessment method to compute assessment scores with some messages will 
already take a few minutes to complete. In a real-world application, this would also 
be unacceptable. 

There is also no complete report printing functionality implemented. This would 
prove to be very useful if a tool of this kind were to be used in a real-educational 


situation. 


The system has plenty of space for further improvements and increases in discussion 


forum features that future researchers might want to follow through: 
L, 


Development of an intelligent student-specific advising module that will give advice 
to each particular student how to improve their participation rating. 

The CMIS could also be improved to integrate a hierarchical level of discussion 
where students start interacting at an easy level topic and slowly progresses to more 
advanced topics. The system should also automatically analyze whether or not 
students can be allowed to participate in higher levels of discussions. 

A multi-group tracking module that tracks student’s participation from one group to 
another. 

Snapshot features where the system will be able to keep a snapshot of the 
participation profiles of students at different stages of the collaboration in order to 


analyze changes in participation/performance results over time. 
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5. The system could be modified to allow the instructor to configure/modify the 
assessment formula by adding and parameterizing other factors such as number of 
contributions, the amount of time spent collaborating, quiz scores, etc. 

6. Printing of reports of student/group assessment could also be implemented as this 


would be a valuable feature, if the tool were to be used in a real-world situation. 
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